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Research studies reflect mixed results on whether or not sendee-learning increases student learning out¬ 
comes. The current study seeks to reconcile these findings by extending a meta-analysis conducted by Novak, 
Markey, and Allen (2007) in which these authors examined sendee-learning and student learning outcomes. 
In the current study, 11 research studies satisfying particular criteria were included. Results suggest that ser¬ 
vice-learning does in fact increase student learning (d = .332). Results from moderator analyses suggest that 
sendee-learning Itas a positive influence on student learning outcomes irrespective of the way learning was 
measured. 


Experiential learning methods, including service¬ 
learning, are increasing among universities across the 
United States. Many disciplines share in this enthusi¬ 
asm, with an increase in academic courses across dis¬ 
ciplines, including communication, using the ser¬ 
vice-learning pedagogy (Oster-Aaland, Sellnow, 
Nelson, & Pearson, 2004; Sellnow & Oster, 1997). 
By definition, service-learning is a pedagogical strat¬ 
egy in which students engage in community service 
that will enhance their understanding of course con¬ 
cepts and enable them to make contributions to their 
communities (Rhodes & Davis, 2001). Furthermore, 
Eyler and Giles (1999) suggest that the service-learn¬ 
ing experience needs to satisfy four criteria to be con¬ 
sidered successful: (1) personal and interpersonal 
development, (2) understanding and applying knowl¬ 
edge learned in class, (3) perspective transformation, 
and (4) developed sense of citizenship. 

A significant amount of research exists regarding 
the positive effects of service-learning on many areas 
of learning including higher order thinking (Eyler & 
Giles, 1999), empathy (Lundy, 2007), cultural aware¬ 
ness (Bloom, 2008; Borden, 2007; Gutheil, 
Chernesky, & Sherratt, 2006), personal and interper¬ 
sonal development (Gullicks, 2006), motivation to 
engage in social issues (Lee, Olszewski-Kubilius, 
Donahue, & Weimholt, 2008), motivation to study 
(Flournoy, 2007), life skills (Astin & Sax, 1998), 
self-efficacy (Simons & Cleary, 2006; Stewart, 
2008), and civic engagement/ responsibility (Astin & 
Sax, 1998; Einfeld & Collins, 2008; Gullicks; Lee et. 
ah; Prentice, 2007; Simons & Cleary). 

However, one important unanswered question 
regarding service-learning is whether this pedagogi¬ 
cal method increases student learning outcomes 
beyond traditional pedagogical methods. Because 
administrators and educators often desire proof that a 


concept "works" or provides significant improve¬ 
ment in student learning over traditional methods to 
support and/or fund such an initiative, several 
researchers have recently attempted to provide evi¬ 
dence that service-learning leads to increased student 
learning. However, the results of these studies are 
mixed. For instance, Kendrick (1996) compared ser¬ 
vice-learning and non-service learning students in an 
Introduction to Sociology course and found that ser¬ 
vice-learning students performed slightly better than 
non-service learning students on their quizzes and 
essay questions. On the other hand, Moely, 
McFarland, Miron, Mercer, and Ilustre (2002) com¬ 
pared service-learning and non-service learning stu¬ 
dents and found that service-learning students report¬ 
ed a slight decrease in learning about the academic 
field over the course of the semester, although it was 
not as large as the decrease shown by students not 
participating in service-learning. 

To reconcile these mixed findings, Novak, Markey, 
and Allen (2007) conducted a meta-analysis evaluat¬ 
ing the cognitive outcomes of service-learning in 
higher education. They examined nine studies and 
found an overall positive relationship between ser¬ 
vice-learning and learning outcomes (d = .424). 
While this meta-analysis provides a significant con¬ 
tribution to the understanding of the relationship 
between service-learning and student learning out¬ 
comes, there were also several problems with this 
meta-analysis. First, Novak et al. did not include 
unpublished literature in their analysis. While not all 
meta-analyses include unpublished data, it is impor¬ 
tant to do so to avoid an upward bias in the effect size 
found (Lipsey & Wilson, 2001). Second, although 
Novak and colleagues found heterogeneity in their 
sample, they did not conduct sufficient moderator 
analyses to explore why their sample was heteroge- 
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neous. For instance, although many of the studies in 
their review examined such concrete measures of 
learning as course exams and/or assignment scores, 
others relied on student and/or faculty self-reported 
learning measures. The difference in the way learn¬ 
ing was measured by the studies included in their 
meta-analysis might have contributed to the hetero¬ 
geneity found in their sample. However, this possi¬ 
bility was not explored in their meta-analysis. 

Scholars have distinguished between concrete mea¬ 
sures of learning and self-reported learning. For exam¬ 
ple, Chesebro and McCroskey (2000) examined the 
correlation between students’ self-reported learning 
and their actual recall of lecture material in a large-lec¬ 
ture communication course. They discovered a moder¬ 
ately significant correlation between self-reported 
learning and actual recall of lecture material (r = -.50, 
p < .001). Although this correlation is significant, if 
self-reported learning measures are accurate measures 
of learning, it seems that the correlation between the 
two should be much stronger. Because student self- 
reported learning can be affected by a number of other 
variables such as teacher immediacy, liking the course 
and/or instructor, and so on, it is important to distin¬ 
guish between these two measurement techniques, 
which Novak et al. (2007) did not do. 

Therefore, the current meta-analysis seeks to build 
on the Novak et al. (2007) meta-analysis so as to fur¬ 
ther clarify the relationship between service-learning 
and student learning outcomes by considering 
unpublished literature and distinguishing between 
self-reported and concrete measures of learning such 
as exams and other assignment scores. Additionally, 
this meta-analysis only examines studies comparing 
service-learning and non-service learning students’ 
learning outcomes. 

Method 

Search Strategy 

To conduct a comprehensive search of studies 
through March 2009, several strategies were 
employed. First, the nine studies included in Novak et 
al.'s (2007) meta-analysis were collected. Next, com¬ 
prehensive searches of electronic databases such as 
Ebscohost and ERIC were conducted. Keywords 
including service-learning and learning outcomes 
were used in this search. Third, because the Michigan 
Journal of Community Service Learning publishes 
service-learning research, a manual search of this 
journal was conducted. 

To include unpublished work related to student 
learning with service-learning, an email message 
soliciting unpublished studies was sent to seven 
prominent researchers in the field, resulting in three 
responses. As a result of one of those responses, a 
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message soliciting unpublished studies was also sent 
to the National Communication Association service¬ 
learning listserv. Finally, a manual search of the 2008 
International Research Conference on Service- 
Learning and Community Engagement conference 
program was conducted. 

All studies identified in the above searches were 
considered for inclusion in this meta-analysis. To 
make the final cut, the studies had to meet the fol¬ 
lowing criteria: (a) examined the relationship 
between service-learning and student learning out¬ 
comes; (b) measured student learning as a dependent 
variable; and (c) included an experimental and con¬ 
trol or comparison group. 

Sample 

A total of 14 studies, including the nine studies 
included in Novak et al.’s (2007) meta-analysis, were 
examined to determine if they met the three criteria. 
Two were excluded because they did not include a 
comparison or control group, and one was excluded 
because it only measured self-efficacy as a depen¬ 
dent variable and not student learning. A final set of 
11 studies meeting the three criteria—including three 
new published studies, two unpublished studies, and 
six included in Novak et al.’s (2007) study—became 
the sample for the present meta-analysis. 

Article Coding 

Articles were coded on many dimensions by the 
researcher. Features coded included demographic 
and sample characteristics as well as intervention and 
methodological characteristics such as the number of 
service hours students were required to complete, the 
type of study design, and the measure of learning 
used. Self-reported and concrete student learning 
outcome measures also were coded for each study. If 
both outcome measures were reported, effect sizes 
were calculated for both measures. 

Effect Size Extraction and Calculation 

Cohen’s d, or the difference between treatment and 
control means divided by the pooled standard devia¬ 
tion, was used as the effect size indicator (Lipsey & 
Wilson, 2001). Effect sizes were calculated using 
data reported in the article (i.e., means and standard 
deviations, f-test, F-test, etc.) using the appropriate 
formulas. In most of the cases, outcome data were 
reported for only one time point. However, in a few 
cases, multiple exam scores were reported (i.e., mid¬ 
semester and final exam scores). In these cases, the 
last exam score of the semester was used to calculate 
the effect size. Additionally, some studies reported 
data for each item measuring student self-reported 
learning. In these cases, an effect size for each item 
was calculated and then averaged to get an overall 
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effect size for that measure. Finally, if insufficient 
data were reported to calculate an effect size, which 
was the case with one study, the article was coded as 
having a zero effect size because there were no sta¬ 
tistically significant differences between the experi¬ 
mental and control groups. 

Meta-Analytic Approach 

Effect sizes were weighted by sample size and 
combined using standard fixed effects meta-analytic 
procedures. The Q statistic was used to examine 
whether significant heterogeneity was present among 
the effect sizes. Effect sizes were calculated for the 
moderator variable discussed earlier—learning out¬ 
come measure. All analyses were conducted using 
Comprehensive Meta-Analysis software. Version 2.0. 

Results 

The 11 studies had a cumulative student sample 
size of 2129 and were published or written (if unpub¬ 
lished) between 1993 and 2008. All studies were con¬ 
ducted using quasi-experimental designs and 
involved undergraduate student samples. A variety of 
course disciplines were represented in these studies, 
including education, English, mass communication, 
pharmacy, political science, psychology, rehabilita¬ 
tion services, and sociology. Of the 11 studies, two 
reported only student self-reported learning out¬ 
comes, two reported both exam scores and student 
self-reported learning, three reported only exam 
scores, two reported other assignment scores, and 
two reported a post-test cognition measure (see Table 


1 for study characteristics). 

Overall Effects of Serx’ice-Leaming on 
Learning Outcomes 

All 11 studies reported learning outcomes, and the 
sample size-weighted mean effect size was d = .332 
(95% Cl = .246, .419; Z= 7.562 ;p = .000; N= 2129). 
This suggests that service-learning has had statisti¬ 
cally significant and positive effects on student learn¬ 
ing outcomes. 

Heterogeneity and Moderators 

Next, heterogeneity was examined. Statistical test¬ 
ing suggested significant heterogeneity among the 
studies, Q = 24.550; p = .017. Therefore, intervention 
moderator analyses were conducted. Both measures 
of learning, including student self-reported learning 
outcomes and more concrete measures of learning 
such as exams and other student assignment scores, 
were significantly and positively related to student 
learning (for self-report: d = .365, p = .000; for 
exams and student assignments: d = .311, p = .000). 
Although, as expected, student self-reported learning 
yielded greater effects on student learning outcomes 
than more concrete measures of learning, there was 
not a statistically significant difference between the 
self-report and concrete measures. Concrete mea¬ 
sures of learning, then, were further divided into sub¬ 
categories, including exam scores, other student 
assignment scores, and post-test measures of cogni¬ 
tion. Once again, as expected, measures of student 
self-reported learning yielded greater effects on 


Table 1 


Characteristics of Studies Included in this Meta-Analysis. 


Study 

Sample 

Discipline 

Outcome Measure Reported 

*Cohen & Kinsey (1994) 

217 undergraduates 

Communication 

Student self-report 

*Feldman et al. (2006) 

32 undergraduates 

English 

Assignment scores 

**Jenkins (2008) 

69 undergraduates 

Political Science 

Post-Test Cognition Measure 

* Kendrick (1996) 

122 undergraduates 

Sociology 

Exam Scores, Student self-report 

*Lundy (2007) 

192 undergraduates 

Psychology 

Exam Scores 

*Markus et al. (1993) 

89 undergraduates 

Political Science 

Exam Scores, Student self-report 

**McIntyre (2008) 

173 undergraduates 

Communication 

Post-Test Cognition Measure 

*Moely et al. (2002) 

536 undergraduates 

Multiple Courses 

Student self-report 

*Mpofu (2007) 

130 undergraduates 

Rehabilitation Services 

Exam Scores 

*Osborne et al. (1998) 

92 undergraduates 

Pharmacy 

Assignment Scores 

*Strage (2000) 

477 undergraduates 

Education 

Exam Scores 


* published manuscript 
** unpublished manuscript 
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learning outcomes than exam scores (for self-report: 
d = .365, p = .000; for exams and student assign¬ 
ments: d = .259, p = .000). However, again, /-tests 
revealed there were no statistically significant differ¬ 
ences between the self-report and concrete measures. 

Discussion 

Consistent with Novak et al.'s (2007) meta-analy- 
sis, the results of the current study suggest that ser¬ 
vice-learning has a positive effect on student learning 
outcomes. This is an encouraging result for educators 
and administrators considering implementing a ser¬ 
vice-learning component into their courses or at their 
universities. Not only does service-learning have 
positive benefits such as increased multicultural 
awareness and enhanced social responsibility, but it 
also increases student learning outcomes, the gold 
standard when measuring pedagogical practices. The 
current study sought to build on Novak et al.'s meta¬ 
analysis by including unpublished work and consid¬ 
ering potential differences between student self- 
reported learning measures and more concrete mea¬ 
sures of learning such as exams and other student 
assignment scores. Although the results of the cur¬ 
rent study suggest that measures of student self- 
reported learning produce greater effects than con¬ 
crete measures of learning, there were no statistically 
significant differences between the two measures. In 
essence, this may suggest that it does not matter how 
student learning is measured. Service-learning likely 
has a positive effect on student learning regardless of 
the learning measure employed. This result also sug¬ 
gests that students have a fairly accurate perception 
of their own learning. 

The current meta-analysis indicates that a service¬ 
learning approach can increase student learning out¬ 
comes. However, the studies examined in this meta¬ 
analysis suffer from problems that can be avoided in 
future research. One significant problem consistent 
across the studies is the lack of discussion regarding 
theory development around service-learning and 
learning outcomes. Although many of the studies 
report the positive influence that a service-learning 
approach has on student learning outcomes, it is not 
clear why this is so. For instance, what is it about ser¬ 
vice-learning that has the ability to influence stu¬ 
dents’ learning of course concepts beyond that of a 
traditionally taught course? Researchers have begun 
to theorize about service-learning and its influence 
on learning outcomes, though the proposed theory 
has not yet been tested (Warren & Sellnow, 2010). 
Warren and Sellnow suggest that theoretical conclu¬ 
sions can be drawn from existing service-learning 
research. For example, research suggests that ser¬ 
vice-learning increases student engagement 
(Rockquemore & Schaffer, 2000) and that when stu¬ 


dents are engaged, they are more motivated to study 
(Flournoy, 2007; Shulman, 1995). As a result, they 
learn more, both cognitively and affectively (Frymier, 
Shulman, & Houser, 1996). 1 However, this theory has 
not been tested; therefore, future researchers might 
consider spending time fleshing out and testing this 
theory. In addition, researchers might develop, test, 
and refine other theories explaining the relationship 
between service-learning and learning outcomes. 

The current study also has limitations. The gener- 
alizability of these results is problematic. Although 
an attempt was made to include unpublished work in 
this meta-analysis, responses to email and listserv 
solicitation only resulted in two unpublished studies. 
It is possible that more unpublished work exists that 
was not included in this study and could potentially 
affect the results. In addition, we were not able to 
gauge the quality of the service-learning experience 
in each of the included studies; quality, of course, is 
an important factor in determining student learning 
outcomes. Next, the studies included in this meta¬ 
analysis only examined college student populations. 
This precludes applying the findings to k-12 stu¬ 
dents. Finally, this meta-analysis only included 11 
total studies. This is largely because many of the 
studies that could have been included did not report 
data that allowed for meta-analysis. 

Therefore, it is important for future researchers to 
include such data when reporting on student learning 
outcomes. First, researchers can include a compari¬ 
son group when examining service-learning out¬ 
comes so that they can be sure the results obtained 
are attributable to the pedagogy and not to some 
other factor. In addition, when reporting outcome 
data on groups (treatment and comparison groups for 
instance), it would be helpful to include means, stan¬ 
dard deviations, and sample sizes of both groups. 
This will allow researchers to compare findings 
across studies and determine overall effect sizes. 

While this study provides us with a positive outlook 
on service-learning, it is important for researchers to 
continue conducting rigorously-designed studies on 
the learning outcomes associated with service-learn¬ 
ing. Additionally, because studies were not found that 
examined populations beyond college students that 
met the inclusion criteria for this meta-analysis, it will 
be important in the future to conduct research with 
elementary, middle, and high school students to deter¬ 
mine service-learning’s effect on them as well. This 
will provide educators and researchers with a more 
comprehensive picture of service-learning's effect on 
the learning of students of all ages. 

Conclusion 

Research on service-learning has shown positive 
effects on many aspects of students' lives including 
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cultural awareness, social responsibility, and student 
cognitive learning outcomes. These are encouraging 
results for advocates of service-learning. Continued 
research on service-learning, especially on popula¬ 
tions beyond college students as well as on theory 
development to explain why researchers and educa¬ 
tors are seeing such promising effects, is still needed. 
But in the meantime, educators can rest easy knowing 
that service-learning projects are likely to have posi¬ 
tive and important learning benefits for their students. 

Note 

1 See the theoretical model outlined in figure 7.1 in 
Warren & Sellnow (2010) for further detail. 
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